Novel Framework of Robot Force Control Using Reinforcement Learning

Over the past decades, robotic technologies have advanced remarkably and have been proven to be successful, especially in the field of manufacturing. In manufacturing, conventional position-controlled robots perform simple repeated tasks in static environments. In recent years, there are increasing needs for robot systems in many areas that involve physical contacts with human-populated environments. Conventional robotic systems, however, have been ineffective in contact tasks. Contrary to robots, humans cope with the problems with dynamic environments by the aid of excellent adaptation and learning ability. In this sense, robot force control strategy inspired by human motor control would be a promising approach. There have been several studies on biologically-inspired motor learning. Cohen et al. suggested impedance learning strategy in a contact task by using associative search network (Cohen et al., 1991). They applied this approach to wall-following task. Another study on motor learning investigated a motor learning method for a musculoskeletal arm model in free space motion using reinforcement learning (Izawa et al., 2002). These studies, however, are limited to rather simple problems. In other studies, artificial neural network models were used for impedance learning in contact tasks (Jung et al., 2001; Tsuji et al., 2004). One of the noticeable works by Tsuji et al. suggested on-line virtual impedance learning method by exploiting visual information. Despite of its usefulness, however, neural network-based learning involves heavy computational load and may lead to local optimum solutions easily. The purpose of this study is to present a novel framework of force control for robotic contact tasks. To develop appropriate motor skills for various contact tasks, this study employs the following methodologies. First, our robot control strategy employs impedance control based on a human motor control theory the equilibrium point control model. The equilibrium point control model suggests that the central nervous system utilizes the springlike property of the neuromuscular system in coordinating multi-DOF human limb movements (Flash, 1987). Under the equilibrium point control scheme, force can be controlled separately by a series of equilibrium points and modulated stiffness (or more generally impedance) at the joints, so the control scheme can become simplified considerably. Second, as the learning framework, reinforcement learning (RL) is employed to optimize the performance of contact task. RL can handle an optimization problem in an